Papers with visual scenes
Multi-Modal Image Captioning for the Visually Impaired (2021.naacl-srw)
Copied to clipboard
| Challenge: | Current captioning models for blind people do not use textual data present in the image when generating captions. |
| Approach: | They propose to use text detected in the image as an input feature in captions . they also use a pointer-generator network to copy detected text to the caption . |
| Outcome: | The proposed system outperforms existing models on the VizWiz dataset, showing a 35% and 16.2% performance improvement. |
Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding (2023.emnlp-demo)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are capable of understanding multi-modal content, but textonly human-computer interaction is not sufficient for many application scenarios. |
| Approach: | They propose a video-to-text generation task and a multi-modal framework that bootstraps cross-modal training from frozen pre-trained visual & audio encoders and frozen LLMs. |
| Outcome: | The proposed framework can understand both visual and auditory content in video and generate meaningful responses grounded in the visual and audio information presented in the videos. |
Visual Evidence Prompting Mitigates Hallucinations in Large Vision-Language Models (2025.acl-long)
Copied to clipboard
| Challenge: | LVLMs have shown impressive progress by integrating visual perception with linguistic understanding to produce contextually grounded outputs. |
| Approach: | They propose a visual evidence prompting method to mitigate hallucinations in large vision-language models by using small visual models to complement them. |
| Outcome: | The proposed method reduces hallucinations by reducing false activation and enhancing correct ones. |
Synchronized Video Storytelling: Generating Video Narrations with Structured Storyline (2024.acl-long)
Copied to clipboard
| Challenge: | Existing studies on dense video captioning and video story generation have made some progress, but in practical applications, we typically require synchronized narrations for ongoing visual scenes. |
| Approach: | They propose a task of Synchronized Video Storytelling to generate synchronized narrations for videos using a benchmark dataset with rich annotations. |
| Outcome: | The proposed framework can generate narrations with the guidance of the generated or predefined storyline and human evaluations validate the effectiveness. |
SceneAlign: Aligning Multimodal Reasoning to Scene Graphs in Complex Visual Scenes (2026.acl-long)
Copied to clipboard
Chuhan Wang, Xintong Li, Jennifer Yuntong Zhang, Junda Wu, Chengkai Huang, Lina Yao, Julian McAuley, Jingbo Shang
| Challenge: | Existing preference-based approaches fail to address this challenge by exploiting language priors to bypass visual grounding. |
| Approach: | They propose a framework that leverages scene graphs as structured visual information to perform controllable structural interventions. |
| Outcome: | The proposed framework improves answer accuracy and reasoning faithfulness across seven visual reasoning benchmarks. |